AFS wine quality project

Introduction

This project attempts to fulfill the assignment outlined in https://www.gastrograph.com/blogs/interviewing-data-science-interns.

“Using only partial least squares regression for generalized linear models (the plsRglm package in R), build an ensemble model to predict the quality score given to each wine from the Vinho Verde region of Portugal (see the data bullet in the Requirements section below to download the datasets). The data consists of chemical tests on wines and an overall assessment score averaged from at least three professional wine tasters. This is interesting data for AFS as the lack of consistent preferences among professional tasters is one of the reasons our company exists.”

“The rubric for assessment is explained in the Selection Criteria section below. The prediction model should be trained and will be tested on both red and white wine after joining the two datasets. We will split the supplied data into an 80% training set and a 20% hold-out validation set before running your training script with a random seed. We will use mean absolute error as a performance metric.”

Load necessary libraries

Several packages needed to be installed in order to use plsRglm. Other packages are loaded for data exploration, tidying, and analysis. The install commands are commented out rather than removed in case they need to be re-installed on another machine.

#install.packages("mvtnorm")

Warning message:
R graphics engine version 14 is not supported by this version of RStudio. The Plots tab will be disabled until a newer version of RStudio is installed. 

#install.packages("bipartite")

# Package 'nloptr' must be installed like this: 
# Download https://github.com/stevengj/nlopt/archive/v2.7.1.tar.gz
# Extract contents, then enter the directory and type the following command:
# cmake . && make && sudo make install
# next type this command:
# sudo apt install libnlopt-dev
# Then you can do this:
#install.packages("nloptr")

# Package 'nloptr' is a prerequisite for 'car'
#install.packages("car")

# Packages 'mvtnorm', 'bipartite', and 'car' are prerequisites for 'plsRglm'.
#install.packages("plsRglm")
library("plsRglm") # See https://arxiv.org/abs/1810.01005 

#install.packages("tidyverse")
library("tidyverse") # for tidying up the data set

Registered S3 methods overwritten by 'dbplyr':
  method         from
  print.tbl_lazy     
  print.tbl_sql      
── Attaching packages ───────────────────────────────────────────── tidyverse 1.3.1 ──
✓ ggplot2 3.3.5     ✓ purrr   0.3.4
✓ tibble  3.1.6     ✓ dplyr   1.0.8
✓ tidyr   1.2.0     ✓ stringr 1.4.0
✓ readr   2.1.2     ✓ forcats 0.5.1
── Conflicts ──────────────────────────────────────────────── tidyverse_conflicts() ──
x dplyr::filter() masks stats::filter()
x dplyr::lag()    masks stats::lag()

#install.packages("ggplot2")
library("ggplot2") # for visualizations
#install.packages("GGally")
library("GGally")

Registered S3 method overwritten by 'GGally':
  method from   
  +.gg   ggplot2

#install.packages('plsdof')
library('plsdof') # Necessary for "plsRglm" to work

Loading required package: MASS

Attaching package: ‘MASS’

The following object is masked from ‘package:dplyr’:

    select

#install.packages("moments")
library('moments') # For calculating skewness of distributions

#install.packages("Metrics")
library("Metrics") # For calculating MAE

library("stringr") # For regex

Read in the data

The data are stored in two files. One contains information on red wine and the other on white wine. In both cases, values are separated by semicolons. We will read in both files, add a variable describing the color of the wine, and then merge them into one dataframe.

# read csv's
df.red <- read.csv("data/winequality-red.csv", sep=';')
df.white <- read.csv("data/winequality-white.csv", sep=';')

# add categorical color variables to both sets
df.red['color'] <- 'red'
df.white['color'] <- 'white'

# merge datasets
df <- rbind(df.white, df.red)

The goal is to use these input variables: 1 - fixed.acidity 2 - volatile.acidity 3 - citric.acid 4 - residual.sugar 5 - chlorides 6 - free.sulfur.dioxide 7 - total.sulfur.dioxide 8 - density 9 - pH 10 - sulphates 11 - alcohol 13 - color

To predict this output variable: 12 - quality (score between 0 and 10)

Explore the data

Let’s print out a few things to get an idea of what the data look like:

str(df)

'data.frame':   6497 obs. of  13 variables:
 $ fixed.acidity       : num  7 6.3 8.1 7.2 7.2 8.1 6.2 7 6.3 8.1 ...
 $ volatile.acidity    : num  0.27 0.3 0.28 0.23 0.23 0.28 0.32 0.27 0.3 0.22 ...
 $ citric.acid         : num  0.36 0.34 0.4 0.32 0.32 0.4 0.16 0.36 0.34 0.43 ...
 $ residual.sugar      : num  20.7 1.6 6.9 8.5 8.5 6.9 7 20.7 1.6 1.5 ...
 $ chlorides           : num  0.045 0.049 0.05 0.058 0.058 0.05 0.045 0.045 0.049 0.044 ...
 $ free.sulfur.dioxide : num  45 14 30 47 47 30 30 45 14 28 ...
 $ total.sulfur.dioxide: num  170 132 97 186 186 97 136 170 132 129 ...
 $ density             : num  1.001 0.994 0.995 0.996 0.996 ...
 $ pH                  : num  3 3.3 3.26 3.19 3.19 3.26 3.18 3 3.3 3.22 ...
 $ sulphates           : num  0.45 0.49 0.44 0.4 0.4 0.44 0.47 0.45 0.49 0.45 ...
 $ alcohol             : num  8.8 9.5 10.1 9.9 9.9 10.1 9.6 8.8 9.5 11 ...
 $ quality             : int  6 6 6 6 6 6 6 6 6 6 ...
 $ color               : chr  "white" "white" "white" "white" ...

head(df)

tail(df)

summary(df)

 fixed.acidity    volatile.acidity  citric.acid     residual.sugar  
 Min.   : 3.800   Min.   :0.0800   Min.   :0.0000   Min.   : 0.600  
 1st Qu.: 6.400   1st Qu.:0.2300   1st Qu.:0.2500   1st Qu.: 1.800  
 Median : 7.000   Median :0.2900   Median :0.3100   Median : 3.000  
 Mean   : 7.215   Mean   :0.3397   Mean   :0.3186   Mean   : 5.443  
 3rd Qu.: 7.700   3rd Qu.:0.4000   3rd Qu.:0.3900   3rd Qu.: 8.100  
 Max.   :15.900   Max.   :1.5800   Max.   :1.6600   Max.   :65.800  
   chlorides       free.sulfur.dioxide total.sulfur.dioxide    density      
 Min.   :0.00900   Min.   :  1.00      Min.   :  6.0        Min.   :0.9871  
 1st Qu.:0.03800   1st Qu.: 17.00      1st Qu.: 77.0        1st Qu.:0.9923  
 Median :0.04700   Median : 29.00      Median :118.0        Median :0.9949  
 Mean   :0.05603   Mean   : 30.53      Mean   :115.7        Mean   :0.9947  
 3rd Qu.:0.06500   3rd Qu.: 41.00      3rd Qu.:156.0        3rd Qu.:0.9970  
 Max.   :0.61100   Max.   :289.00      Max.   :440.0        Max.   :1.0390  
       pH          sulphates         alcohol         quality         color          
 Min.   :2.720   Min.   :0.2200   Min.   : 8.00   Min.   :3.000   Length:6497       
 1st Qu.:3.110   1st Qu.:0.4300   1st Qu.: 9.50   1st Qu.:5.000   Class :character  
 Median :3.210   Median :0.5100   Median :10.30   Median :6.000   Mode  :character  
 Mean   :3.219   Mean   :0.5313   Mean   :10.49   Mean   :5.818                     
 3rd Qu.:3.320   3rd Qu.:0.6000   3rd Qu.:11.30   3rd Qu.:6.000                     
 Max.   :4.010   Max.   :2.0000   Max.   :14.90   Max.   :9.000                     

Let’s now start exploring the data one variable at a time. I’ve created a function below that will make a plot of two overlapping transparent histograms. It will be useful to compare the distributions for red wine and white wine.

plot_multi_histogram <- function(df, feature, label.column, alpha, binwidth) {
  plt <- ggplot(df, aes(x=eval(parse(text=feature)), fill=eval(parse(text=label.column)))) +
    geom_histogram(alpha=alpha, position="identity", color="black", binwidth=binwidth) +
  #geom_density(alpha=alpha)# +
  #geom_vline(aes(xintercept=mean(eval(parse(text=feature)))), color="black", linetype="dashed", size=1)# +
  labs(x=feature, y = "Density")
  plt + guides(fill=guide_legend(title=label.column))
}

fixed.acidity

#   1 - fixed acidity
summary(df$fixed.acidity)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  3.800   6.400   7.000   7.215   7.700  15.900 

#table(df$fixed.acidity)
#sort(unique(df$fixed.acidity))

#ggplot(data = df, aes(x=fixed.acidity, fill=color)) + geom_histogram(alpha=0.2, position="identity")
plot_multi_histogram(df, "fixed.acidity", "color", alpha=0.4, binwidth=1)

volatile.acidity

#   2 - volatile acidity
summary(df$volatile.acidity)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.0800  0.2300  0.2900  0.3397  0.4000  1.5800 

plot_multi_histogram(df, "volatile.acidity", "color", alpha=0.4, binwidth=0.1)

citric.acid

#   3 - citric acid
summary(df$citric.acid)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.0000  0.2500  0.3100  0.3186  0.3900  1.6600 

plot_multi_histogram(df, "citric.acid", "color", alpha=0.4, binwidth=0.1)

residual.sugar

#   4 - residual sugar
summary(df$residual.sugar)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  0.600   1.800   3.000   5.443   8.100  65.800 

plot_multi_histogram(df, "residual.sugar", "color", alpha=0.4, binwidth=2)

chlorides

#   5 - chlorides
summary(df$chlorides)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.00900 0.03800 0.04700 0.05603 0.06500 0.61100 

plot_multi_histogram(df, "chlorides", "color", alpha=0.4, binwidth=0.01)

free.sulfur.dioxide

#   6 - free sulfur dioxide
summary(df$free.sulfur.dioxide)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   1.00   17.00   29.00   30.53   41.00  289.00 

plot_multi_histogram(df, "free.sulfur.dioxide", "color", alpha=0.4, binwidth=10)

total.sulfur.dioxide

#   7 - total sulfur dioxide
summary(df$total.sulfur.dioxide)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    6.0    77.0   118.0   115.7   156.0   440.0 

plot_multi_histogram(df, "total.sulfur.dioxide", "color", alpha=0.4, binwidth=10)

density

#   8 - density
summary(df$density)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.9871  0.9923  0.9949  0.9947  0.9970  1.0390 

plot_multi_histogram(df, "density", "color", alpha=0.4, binwidth=0.001)

pH

#   9 - pH
summary(df$pH)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  2.720   3.110   3.210   3.219   3.320   4.010 

plot_multi_histogram(df, "pH", "color", alpha=0.4, binwidth=0.05)

sulphates

#   10 - sulphates
summary(df$sulphates)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.2200  0.4300  0.5100  0.5313  0.6000  2.0000 

plot_multi_histogram(df, "sulphates", "color", alpha=0.4, binwidth=0.1)

alcohol

#   11 - alcohol
summary(df$alcohol)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   8.00    9.50   10.30   10.49   11.30   14.90 

plot_multi_histogram(df, "alcohol", "color", alpha=0.4, binwidth=0.2)

quality

#   12 - quality (score between 0 and 10)
summary(df$quality)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  3.000   5.000   6.000   5.818   6.000   9.000 

plot_multi_histogram(df, "quality", "color", alpha=0.4, binwidth=1)

By looking at the data, we have learned the following:

There is more data on white wine than red wine, so we may need to rescale the data so that the model learns to classify red wine as well as white wine.

Several of the distributions are skewed, indicating the presence of long tails or outliers.

Now let’s look for correlations between features in the data with a ggpairs plot.

ggpairs(df, columns=1:12, mapping=ggplot2::aes(colour = color), lower=list(continuous='points')) + scale_color_manual(values=c("red", "yellow"))

The highest correlations we see are the following:

0.721 between free.sulfur.dioxide and total.sulfur.dioxide

0.687 between density and alcohol

0.553 between density and residual.sugar

0.495 between total.sulfur.dioxide and residual.sugar

0.459 between density and fixed.acidity

0.444 between alcohol and quality

0.414 between total.sulfur.dioxide and volatile.acidity

0.403 between free.sulfur.dioxide and residual.sugar

There is also a correlation between pH and volatile.acidity, but it’s different between red wine and white wine.

Cleary the alcohol content will be useful for our model because of its relatively high correlation with the wine quality rating (0.444).

We also see some outliers. Let’s take a closer look at some of the plots to figure out where these outliers are and if they ought to be removed. The most obvious outlier is the one in residual.sugar and density.

ggplot(df, aes(x=residual.sugar, y=density, color=as.factor(quality))) +
  geom_point(size=6) + scale_color_brewer(palette = "Spectral")

The maximum has over twice as much sugar as the next largest point. It is also twice as far from the minimum density as the next most dense point. It is clearly an outlier and I believe it is safe to remove it from the data. We’ll do this by cutting out all points with residual.sugar > 40. The next largest point is also separated from most of the other points, but more so in density than in residual.sugar. The density is correlated with both alcohol (0.687) and residual.sugar (0.553). I think we can assume that much of the information about the density is included already in the alcohol and sugar variables, so we can remove density from the dataset.

Another outlier is in free.sulfur.dioxide and total.sulfur.dioxide.

ggplot(df, aes(x=free.sulfur.dioxide, y=total.sulfur.dioxide, color=as.factor(quality))) +
  geom_point(size=6) + scale_color_brewer(palette = "Spectral")

This point has almost twice as much free sulfur dioxide as the next largest point. However, it is not much of an outlier on the other axis, total.sulfur.dioxide. It also has the minimum quality rating, which means it could have useful predictive power. There is a strong correlation between these two variables (0.721), which means it may be better to simply remove the variable free.sulfur.dioxide from the dataset rather than remove this data point.

Let’s apply the changes (remove sugar outlier & remove free.sulfur.dioxide (6) and density (8)).

df.cut <- df[df$residual.sugar < 40, c(1:5, 7, 9:13)]

Now we’ll redraw the plots.

ggpairs(df.cut, columns=1:10, mapping=ggplot2::aes(colour = color), lower=list(continuous='points')) + scale_color_manual(values=c("red", "yellow"))

There are still some points that tend to isolate themselves from most of the data, but not to such a degree as the outlier that we removed. They seem to simply follow the shape of a long-tailed distribution. This calls for a transformation of the data to make the distributions more normal.

Let’s begin the transformations by first calculating the skewness of the variables.

for (name in names(df.cut[1:9])) {
  skew <- skewness(df.cut[name], na.rm = TRUE)
  print(paste(name,skew))
}

[1] "fixed.acidity 1.72303425219973"
[1] "volatile.acidity 1.49292242490666"
[1] "citric.acid 0.47177011412896"
[1] "residual.sugar 1.1695985102309"
[1] "chlorides 5.39870943969405"
[1] "total.sulfur.dioxide -0.000889571569577466"
[1] "pH 0.387127126904609"
[1] "sulphates 1.79749476995869"
[1] "alcohol 0.565985435318878"

We now create a copy of the dataframe and apply transformations. The functions 1/x, sqrt, and log10 are useful for transforming positively skewed distributions into a more gaussian shape, depending on the severity of the skew. The following transformations give decent distributions.

df.transformed <- data.frame(df.cut)
df.transformed$fixed.acidity <- 1/(df.transformed$fixed.acidity)
df.transformed$sulphates <- 1/(df.transformed$sulphates)
df.transformed$volatile.acidity <- sqrt(1/(df.transformed$volatile.acidity))
df.transformed$residual.sugar <- log10(df.transformed$residual.sugar)
df.transformed$chlorides <- sqrt(1/(df.transformed$chlorides))
df.transformed$citric.acid <- sqrt(df.transformed$citric.acid + 0.1)
df.transformed$total.sulfur.dioxide <- sqrt(df.transformed$total.sulfur.dioxide)

Let’s plot a few histograms to see how normal the new distributions are.

plot_multi_histogram(df.transformed, "volatile.acidity", "color", alpha=0.4, binwidth=0.1)

plot_multi_histogram(df.transformed, "residual.sugar", "color", alpha=0.4, binwidth=0.1)

plot_multi_histogram(df.transformed, "chlorides", "color", alpha=0.4, binwidth=2)

plot_multi_histogram(df.transformed, "citric.acid", "color", alpha=0.4, binwidth=0.05)

We can now calculate the skewness again to see if it has improved.

for (name in names(df.transformed[1:9])) {
  skew <- skewness(df.transformed[name], na.rm = TRUE)
  print(paste(name,skew))
}

[1] "fixed.acidity -0.158566250269554"
[1] "volatile.acidity 0.248389283010552"
[1] "citric.acid -0.444243638771569"
[1] "residual.sugar 0.237148189575897"
[1] "chlorides 0.115471194185833"
[1] "total.sulfur.dioxide -0.652244809027631"
[1] "pH 0.387127126904609"
[1] "sulphates 0.441836626677462"
[1] "alcohol 0.565985435318878"

We can also plot a few scatter plots to look at two variables at once, to see how the distributions changed after the transformations and look for any more correlations. We can color-code the quality score.

ggplot(df.cut, aes(x=fixed.acidity, y=sulphates, color=as.factor(quality))) +
  geom_point(size=6) + scale_color_brewer(palette = "Spectral")

ggplot(df.transformed, aes(x=fixed.acidity, y=sulphates, color=as.factor(quality))) +
  geom_point(size=6) + scale_color_brewer(palette = "Spectral")

In this particular case it looks like the data have become more normally distributed. Let’s look at the full correlation plot again because it gives a good view of the entire dataset all at once.

ggpairs(df.transformed, columns=1:10, mapping=ggplot2::aes(colour = color), lower=list(continuous='points')) + scale_color_manual(values=c("red", "yellow"))

The data look very good now. However, there does not seem to be any easy way to predict the quality of the wine. None of the variables have a very high correlation with quality. The colored scatter plots that we have looked at do not have any obvious clustering of quality scores. Any significant correlations with quality must reside in combinations of more that two features, in a very nonlinear way. This makes the required application of a linear model unusual, but let’s see what we can do.

First model attempt

The data appear to be ready for the application of a simple test model to see if we can get the plsRglm package working correctly. Let’s transform the color variable into numeric factors and set up our X variables and Y target.

dataY <- factor(df.transformed$quality, ordered=TRUE)
dataX <- df.transformed[1:9]
df.transformed$color <- factor(df.transformed$color)
dataX['color'] <- as.numeric(df.transformed$color)

As stated previously, the data ought to be weighted by color so that the model isn’t biased towards white wine. We want it to learn to accurately predict the quality of either color. In this case, the difference in size of the two groups is not extreme, but it is still good practice to be aware of and take care of these issues.

weight.factor <- nrow(df.red)/nrow(df.white)
get_weight <- function(color.name) {
  if (color.name == "white") {
    return(weight.factor)
  }
  return(1.0)
}
prior.weights <- sapply(df.transformed$color, get_weight)

We now build a model using the required package plsRglm. See https://cran.r-project.org/web/packages/plsRglm/vignettes/plsRglm.pdf for more information on this package and specific applications. More information about the arguments can be found at https://www.rdocumentation.org/packages/plsRglm/versions/1.3.0/topics/plsRglm. We will train on 80% of the data and save 20% for testing later.

N.rows <- nrow(dataX)
train.fraction <- 0.8
N.train.rows <- floor(N.rows*train.fraction)
N.test.rows <- N.rows - N.train.rows
test.indices <- sample.int(N.rows, N.test.rows) # select N_test_rows randomly
dataX.test <- dataX[c(test.indices),]
dataX.train <- dataX[-c(test.indices),]
dataY.test <- dataY[c(test.indices)]
dataY.train <- dataY[-c(test.indices)]
prior.weights.test <- prior.weights[c(test.indices)]
prior.weights.train <- prior.weights[-c(test.indices)]

set.seed(123)
model.pls <- plsRglm(dataY.train,dataX.train,nt=10,limQ2set=.0975,
             dataPredictY=dataX.train,modele="pls-glm-polr",family=NULL,typeVC="none",
             EstimXNA=FALSE,scaleX=TRUE,scaleY=NULL,pvals.expli=FALSE,
             alpha.pvals.expli=.05,MClassed=FALSE,tol_Xi=10^(-12),weights=prior.weights.train,
             sparse=FALSE,sparseStop=TRUE,naive=FALSE,verbose=TRUE)

____************************************************____
Only naive DoF can be used with PLS GLM
Only naive DoF can be used with weighted PLS

Model: pls-glm-polr 
Method: logistic 

____Component____ 1 ____
____Component____ 2 ____
____Component____ 3 ____
____Component____ 4 ____
____Component____ 5 ____
No more significant predictors (<0.05) found
Warning only 5 components were thus extracted
____Predicting X without NA neither in X nor in Y____
****________________________________________________****

model.pls

Number of required components:
[1] 10
Number of successfully computed components:
[1] 5
Coefficients:
                            [,1]
3|4                   7.43119294
4|5                   9.54260785
5|6                  12.61849044
6|7                  15.21585817
7|8                  17.57477370
8|9                  21.60134406
fixed.acidity        -1.97867791
volatile.acidity     -1.67664252
citric.acid          -0.45936171
residual.sugar       -1.01052214
chlorides            -0.24314098
total.sulfur.dioxide  0.02653666
pH                   -0.33875058
sulphates             0.50649502
alcohol              -0.88540091
color                 0.60749011
Information criteria and Fit statistics:
               AIC      BIC Missclassed Chi2_Pearson_Y
Nb_Comp_0 13287.31 13326.65        2928       6708.064
Nb_Comp_1 11936.06 11981.95        2579       5155.807
Nb_Comp_2 11634.73 11687.17        2437       4866.270
Nb_Comp_3 11486.87 11545.87        2419       4714.235
Nb_Comp_4 11435.80 11501.36        2377       4672.224
Nb_Comp_5 11431.05 11503.16        2370       4660.304

Let’s take a look at how well the model performs in predicting the quality of wines. We compare the predictions and observations with a table, a plot, and a calculation of the Mean Absolute Error (MAE).

predictions.train <- as.numeric(predict(model.pls$FinalModel))
observations.train <- as.numeric(dataY.train)
predictions.test <- as.numeric(predict(model.pls, newdata = dataX.test, type="class"))
observations.test <- as.numeric(dataY.test)
table(as.numeric(unlist(dataY.train)), predictions.train )

   predictions.train
       3    4    5    6
  1   16   11    0    0
  2   99   76    5    0
  3 1008  676   15    1
  4  477 1650  141    0
  5   36  660  168    2
  6    7  101   44    0
  7    0    2    1    0

table(as.numeric(unlist(dataY.test)), predictions.test )

   predictions.test
      3   4   5
  1   2   1   0
  2  22  14   0
  3 254 183   1
  4 130 393  44
  5  10 162  41
  6   3  27  11
  7   0   0   2

plot(observations.test, predictions.test)

mae(observations.train, predictions.train)

[1] 0.5132794

mae(observations.test, predictions.test)

[1] 0.52

This is not bad. It’s much better than randomly guessing numbers between 3 and 7, in which case I would expect an mae of about 1.8. But still, maybe we can do better.

A cross validation experiment demonstrates that 10 is an appropriate number of components for this model, given the training data. However, this takes a very long time to run. So I’m excluding it from this notebook.

#cv.modpls <- cv.plsRglm(dataY.train,dataX.train,nt=10,modele="pls-glm-polr",NK=5)
#cv.modpls
#res.cv.modpls=cvtable(summary(cv.modpls, MClassed = TRUE))
#plot(res.cv.modpls) # This suggests that 10 is the preferred number of components for the model.

Ensemble

Let’s try to improve the model by implementing an ensemble. Linear regression models are very robust and tend to get the same result every time, so making the same model again and again and averaging the results won’t do any good, unlike a random forest which can create thousands of different trees by simply changing the random see. Instead of bagging models together, let’s try a boosting technique. The first model will use the features to try and predict the quality score. The second model will use the features and the 1st model’s score to try and predict the true quality score, thus fitting to the difference, and so on.

dataX.train.2 <- data.frame(dataX.train)
dataX.train.2["prediction1"] <- as.numeric( predict(model.pls$FinalModel) )
dataX.test.2 <- data.frame(dataX.test)
dataX.test.2["prediction1"] <- as.numeric( predict(model.pls, newdata = dataX.test, type="class") )
model.pls.2 <- plsRglm(dataY.train,dataX.train.2,nt=10,limQ2set=.0975,
                     dataPredictY=dataX.train.2,modele="pls-glm-polr",family=NULL,typeVC="none",
                     EstimXNA=FALSE,scaleX=TRUE,scaleY=NULL,pvals.expli=FALSE,
                     alpha.pvals.expli=.05,MClassed=FALSE,tol_Xi=10^(-12),weights=prior.weights.train)

____************************************************____
Only naive DoF can be used with weighted PLS

Model: pls-glm-polr 
Method: logistic 

____Component____ 1 ____
____Component____ 2 ____
____Component____ 3 ____
____Component____ 4 ____
____Component____ 5 ____
____Component____ 6 ____
____Component____ 7 ____
____Component____ 8 ____
____Component____ 9 ____
____Component____ 10 ____
____Predicting X without NA neither in X nor in Y____
****________________________________________________****

model.pls.2

Number of required components:
[1] 10
Number of successfully computed components:
[1] 10
Coefficients:
                            [,1]
3|4                   6.87068810
4|5                   8.98709904
5|6                  12.06270840
6|7                  14.66104119
7|8                  17.02131062
8|9                  21.04616144
fixed.acidity        -1.75316521
volatile.acidity     -1.66815402
citric.acid          -0.33562439
residual.sugar       -0.98455757
chlorides            -0.22967034
total.sulfur.dioxide  0.01493810
pH                   -0.15275462
sulphates             0.49511138
alcohol              -0.89712185
color                 0.65638385
prediction1          -0.01754957
Information criteria and Fit statistics:
                AIC      BIC Missclassed Chi2_Pearson_Y
Nb_Comp_0  13287.31 13326.65        2928       6708.064
Nb_Comp_1  11737.32 11783.21        2391       4675.422
Nb_Comp_2  11606.70 11659.15        2371       4383.452
Nb_Comp_3  11496.50 11555.50        2378       4415.080
Nb_Comp_4  11431.92 11497.48        2377       4681.173
Nb_Comp_5  11429.96 11502.07        2371       4646.434
Nb_Comp_6  11431.47 11510.14        2368       4653.847
Nb_Comp_7  11433.39 11518.62        2372       4647.880
Nb_Comp_8  11435.39 11527.17        2374       4649.227
Nb_Comp_9  11437.38 11535.72        2372       4649.341
Nb_Comp_10 11439.38 11544.27        2371       4649.378

We now calculate the MAE to see how well the second model performed.

predictions.train <- as.numeric(predict(model.pls.2$FinalModel))
observations.train <- as.numeric(dataY.train)
predictions.test <- as.numeric(predict(model.pls.2, newdata = dataX.test.2, type="class"))
observations.test <- as.numeric(dataY.test)
mae(observations.train, predictions.train)

[1] 0.5134719

mae(observations.test, predictions.test)

[1] 0.5161538

Boosting in this manner offered no significant improvement in the MAE.

We could try a boosted ensemble where we just do one variable at a time and then fit the prediction error with each successive model, but I think that would be just like multivariate regression, but not as good.

Let’s continue trying a boosted ensemble with all of the variables, including the result of the previous model to make the next predictions. But instead of trying to fit the quality values at each step, we can try to fit the difference between the quality values and the previous prediction. Let’s try it.

number.of.models <- 3
models.pls <- vector(mode = "list", length = number.of.models) # empty list of models
errors <- vector(mode = "list", length = number.of.models) # empty list of errors
predictions <- as.numeric(dataY.train)*0.0
targets <- as.numeric(dataY.train)
for (i in 1:number.of.models) {
  dataX.this <- data.frame(dataX.train)
  if (i > 1) {
    dataX.this["prediction1"] <- predictions
  }
  targets <- targets - predictions
  model.this <- plsRglm(targets,dataX.this,nt=10,limQ2set=.0975,
                       dataPredictY=dataX.this,modele="pls-glm-polr",family=NULL,typeVC="none",
                       EstimXNA=FALSE,scaleX=TRUE,scaleY=NULL,pvals.expli=FALSE,
                       alpha.pvals.expli=.05,MClassed=FALSE,tol_Xi=10^(-12),weights=prior.weights.train,
                       sparse=FALSE,sparseStop=TRUE,naive=FALSE,verbose=TRUE)
  #model_this
  predictions <- as.numeric(predict(model.this$FinalModel))
  observations <- as.numeric(targets)
  errors[i] <- mae(observations, predictions)
  models_pls[i] <- model_this
  print(errors[i])
}

____************************************************____
Only naive DoF can be used with PLS GLM
Only naive DoF can be used with weighted PLS

Model: pls-glm-polr 
Method: logistic 

____Component____ 1 ____
____Component____ 2 ____
____Component____ 3 ____
____Component____ 4 ____
____Component____ 5 ____
No more significant predictors (<0.05) found
Warning only 5 components were thus extracted
____Predicting X without NA neither in X nor in Y____
****________________________________________________****

number of items to replace is not a multiple of replacement length

[[1]]
[1] 0.5132794

____************************************************____
Only naive DoF can be used with PLS GLM
Only naive DoF can be used with weighted PLS

Model: pls-glm-polr 
Method: logistic 

____Component____ 1 ____
____Component____ 2 ____
____Component____ 3 ____
____Component____ 4 ____
____Component____ 5 ____
____Component____ 6 ____
No more significant predictors (<0.05) found
Warning only 6 components were thus extracted
____Predicting X without NA neither in X nor in Y____
****________________________________________________****

number of items to replace is not a multiple of replacement length

[[1]]
[1] 3.961124

____************************************************____
Only naive DoF can be used with PLS GLM
Only naive DoF can be used with weighted PLS

Model: pls-glm-polr 
Method: logistic 

____Component____ 1 ____
____Component____ 2 ____
No more significant predictors (<0.05) found
Warning only 2 components were thus extracted
____Predicting X without NA neither in X nor in Y____
****________________________________________________****

number of items to replace is not a multiple of replacement length

[[1]]
[1] 7.946112

This is resulting in models that get progressively worse. I see now that I was on the wrong track. The data are too noisy, so by boosting a noisy model I just get an even noisier model. What I can do is bag the models instead, which is what I didn’t want to do at first because every model would be the same, but actually that’s only true if I use the same data each time. I could split up the dataset into N chunks and fit each one separately, and then I’ll use the results all together for the final prediction. But that’s basically just multivariate linear regression on the full dataset, only worse.

Engineering cross-terms for effective polynomial fitting

Let’s try another approach. Since the plsRglm model is designed to handle high dimensional data with few rows, I’m going to increase the dimensionality of the dataset. This will allow previously nonlinear quadratic properties to be described linearly by the plsRglm model.

If I think that the relationship between the data and quality is nonlinear then I just need to create nonlinear features. If I want xy in the model instead of just ax + by I can make a column xy by multiplying two columns together. I’ll multiply every continuous feature by every other continuous feature to get new quadratic columns.

dataX.nonlinear <- data.frame(dataX)
for (feature1 in names(dataX.nonlinear[1:9])) {
  for (feature2 in names(dataX.nonlinear[1:9])) {
    new.feature <- paste(substring(feature1,1,2), substring(feature2,1,2), sep=".")
    dataX.nonlinear[new.feature] <- dataX.nonlinear[feature1]*dataX.nonlinear[feature2]
  }
}

names(dataX_nonlinear)

  [1] "fixed.acidity"          "volatile.acidity"       "citric.acid"           
  [4] "residual.sugar"         "chlorides"              "total.sulfur.dioxide"  
  [7] "pH"                     "sulphates"              "alcohol"               
 [10] "color"                  "fi.fi"                  "fi.vo"                 
 [13] "fi.ci"                  "fi.re"                  "fi.ch"                 
 [16] "fi.to"                  "fi.pH"                  "fi.su"                 
 [19] "fi.al"                  "vo.fi"                  "vo.vo"                 
 [22] "vo.ci"                  "vo.re"                  "vo.ch"                 
 [25] "vo.to"                  "vo.pH"                  "vo.su"                 
 [28] "vo.al"                  "ci.fi"                  "ci.vo"                 
 [31] "ci.ci"                  "ci.re"                  "ci.ch"                 
 [34] "ci.to"                  "ci.pH"                  "ci.su"                 
 [37] "ci.al"                  "re.fi"                  "re.vo"                 
 [40] "re.ci"                  "re.re"                  "re.ch"                 
 [43] "re.to"                  "re.pH"                  "re.su"                 
 [46] "re.al"                  "ch.fi"                  "ch.vo"                 
 [49] "ch.ci"                  "ch.re"                  "ch.ch"                 
 [52] "ch.to"                  "ch.pH"                  "ch.su"                 
 [55] "ch.al"                  "to.fi"                  "to.vo"                 
 [58] "to.ci"                  "to.re"                  "to.ch"                 
 [61] "to.to"                  "to.pH"                  "to.su"                 
 [64] "to.al"                  "pH.fi"                  "pH.vo"                 
 [67] "pH.ci"                  "pH.re"                  "pH.ch"                 
 [70] "pH.to"                  "pH.pH"                  "pH.su"                 
 [73] "pH.al"                  "su.fi"                  "su.vo"                 
 [76] "su.ci"                  "su.re"                  "su.ch"                 
 [79] "su.to"                  "su.pH"                  "su.su"                 
 [82] "su.al"                  "al.fi"                  "al.vo"                 
 [85] "al.ci"                  "al.re"                  "al.ch"                 
 [88] "al.to"                  "al.pH"                  "al.su"                 
 [91] "al.al"                  "r.fixed.acidity"        "w.fixed.acidity"       
 [94] "r.volatile.acidity"     "w.volatile.acidity"     "r.citric.acid"         
 [97] "w.citric.acid"          "r.residual.sugar"       "w.residual.sugar"      
[100] "r.chlorides"            "w.chlorides"            "r.total.sulfur.dioxide"
[103] "w.total.sulfur.dioxide" "r.pH"                   "w.pH"                  
[106] "r.sulphates"            "w.sulphates"            "r.alcohol"             
[109] "w.alcohol"              "r.fi.fi"                "w.fi.fi"               
[112] "r.fi.vo"                "w.fi.vo"                "r.fi.ci"               
[115] "w.fi.ci"                "r.fi.re"                "w.fi.re"               
[118] "r.fi.ch"                "w.fi.ch"                "r.fi.to"               
[121] "w.fi.to"                "r.fi.pH"                "w.fi.pH"               
[124] "r.fi.su"                "w.fi.su"                "r.fi.al"               
[127] "w.fi.al"                "r.vo.fi"                "w.vo.fi"               
[130] "r.vo.vo"                "w.vo.vo"                "r.vo.ci"               
[133] "w.vo.ci"                "r.vo.re"                "w.vo.re"               
[136] "r.vo.ch"                "w.vo.ch"                "r.vo.to"               
[139] "w.vo.to"                "r.vo.pH"                "w.vo.pH"               
[142] "r.vo.su"                "w.vo.su"                "r.vo.al"               
[145] "w.vo.al"                "r.ci.fi"                "w.ci.fi"               
[148] "r.ci.vo"                "w.ci.vo"                "r.ci.ci"               
[151] "w.ci.ci"                "r.ci.re"                "w.ci.re"               
[154] "r.ci.ch"                "w.ci.ch"                "r.ci.to"               
[157] "w.ci.to"                "r.ci.pH"                "w.ci.pH"               
[160] "r.ci.su"                "w.ci.su"                "r.ci.al"               
[163] "w.ci.al"                "r.re.fi"                "w.re.fi"               
[166] "r.re.vo"                "w.re.vo"                "r.re.ci"               
[169] "w.re.ci"                "r.re.re"                "w.re.re"               
[172] "r.re.ch"                "w.re.ch"                "r.re.to"               
[175] "w.re.to"                "r.re.pH"                "w.re.pH"               
[178] "r.re.su"                "w.re.su"                "r.re.al"               
[181] "w.re.al"                "r.ch.fi"                "w.ch.fi"               
[184] "r.ch.vo"                "w.ch.vo"                "r.ch.ci"               
[187] "w.ch.ci"                "r.ch.re"                "w.ch.re"               
[190] "r.ch.ch"                "w.ch.ch"                "r.ch.to"               
[193] "w.ch.to"                "r.ch.pH"                "w.ch.pH"               
[196] "r.ch.su"                "w.ch.su"                "r.ch.al"               
[199] "w.ch.al"                "r.to.fi"                "w.to.fi"               
[202] "r.to.vo"                "w.to.vo"                "r.to.ci"               
[205] "w.to.ci"                "r.to.re"                "w.to.re"               
[208] "r.to.ch"                "w.to.ch"                "r.to.to"               
[211] "w.to.to"                "r.to.pH"                "w.to.pH"               
[214] "r.to.su"                "w.to.su"                "r.to.al"               
[217] "w.to.al"                "r.pH.fi"                "w.pH.fi"               
[220] "r.pH.vo"                "w.pH.vo"                "r.pH.ci"               
[223] "w.pH.ci"                "r.pH.re"                "w.pH.re"               
[226] "r.pH.ch"                "w.pH.ch"                "r.pH.to"               
[229] "w.pH.to"                "r.pH.pH"                "w.pH.pH"               
[232] "r.pH.su"                "w.pH.su"                "r.pH.al"               
[235] "w.pH.al"                "r.su.fi"                "w.su.fi"               
[238] "r.su.vo"                "w.su.vo"                "r.su.ci"               
[241] "w.su.ci"                "r.su.re"                "w.su.re"               
[244] "r.su.ch"                "w.su.ch"                "r.su.to"               
[247] "w.su.to"                "r.su.pH"                "w.su.pH"               
[250] "r.su.su"                "w.su.su"                "r.su.al"               
[253] "w.su.al"                "r.al.fi"                "w.al.fi"               
[256] "r.al.vo"                "w.al.vo"                "r.al.ci"               
[259] "w.al.ci"                "r.al.re"                "w.al.re"               
[262] "r.al.ch"                "w.al.ch"                "r.al.to"               
[265] "w.al.to"                "r.al.pH"                "w.al.pH"               
[268] "r.al.su"                "w.al.su"                "r.al.al"               
[271] "w.al.al"               

I should also separate the red and white wine features, because they appear to be significantly different.

for (feature in names(dataX.nonlinear[c(1:9, 11:91)])) {
  
  r.feature <- paste("r", feature, sep=".")
  w.feature <- paste("w", feature, sep=".")
  dataX.nonlinear[r.feature] <- with(dataX.nonlinear, 
                                        ifelse(color == 1,
                                               eval(parse(text=feature)),
                                               0.0)
                                        )
  dataX.nonlinear[w.feature] <- with(dataX.nonlinear, 
                                        ifelse(color == 2,
                                               eval(parse(text=feature)),
                                               0.0)
                                        )
}

Let’s look at the new dataframe features.

names(dataX_nonlinear)

  [1] "fixed.acidity"          "volatile.acidity"       "citric.acid"           
  [4] "residual.sugar"         "chlorides"              "total.sulfur.dioxide"  
  [7] "pH"                     "sulphates"              "alcohol"               
 [10] "color"                  "fi.fi"                  "fi.vo"                 
 [13] "fi.ci"                  "fi.re"                  "fi.ch"                 
 [16] "fi.to"                  "fi.pH"                  "fi.su"                 
 [19] "fi.al"                  "vo.fi"                  "vo.vo"                 
 [22] "vo.ci"                  "vo.re"                  "vo.ch"                 
 [25] "vo.to"                  "vo.pH"                  "vo.su"                 
 [28] "vo.al"                  "ci.fi"                  "ci.vo"                 
 [31] "ci.ci"                  "ci.re"                  "ci.ch"                 
 [34] "ci.to"                  "ci.pH"                  "ci.su"                 
 [37] "ci.al"                  "re.fi"                  "re.vo"                 
 [40] "re.ci"                  "re.re"                  "re.ch"                 
 [43] "re.to"                  "re.pH"                  "re.su"                 
 [46] "re.al"                  "ch.fi"                  "ch.vo"                 
 [49] "ch.ci"                  "ch.re"                  "ch.ch"                 
 [52] "ch.to"                  "ch.pH"                  "ch.su"                 
 [55] "ch.al"                  "to.fi"                  "to.vo"                 
 [58] "to.ci"                  "to.re"                  "to.ch"                 
 [61] "to.to"                  "to.pH"                  "to.su"                 
 [64] "to.al"                  "pH.fi"                  "pH.vo"                 
 [67] "pH.ci"                  "pH.re"                  "pH.ch"                 
 [70] "pH.to"                  "pH.pH"                  "pH.su"                 
 [73] "pH.al"                  "su.fi"                  "su.vo"                 
 [76] "su.ci"                  "su.re"                  "su.ch"                 
 [79] "su.to"                  "su.pH"                  "su.su"                 
 [82] "su.al"                  "al.fi"                  "al.vo"                 
 [85] "al.ci"                  "al.re"                  "al.ch"                 
 [88] "al.to"                  "al.pH"                  "al.su"                 
 [91] "al.al"                  "r.fixed.acidity"        "w.fixed.acidity"       
 [94] "r.volatile.acidity"     "w.volatile.acidity"     "r.citric.acid"         
 [97] "w.citric.acid"          "r.residual.sugar"       "w.residual.sugar"      
[100] "r.chlorides"            "w.chlorides"            "r.total.sulfur.dioxide"
[103] "w.total.sulfur.dioxide" "r.pH"                   "w.pH"                  
[106] "r.sulphates"            "w.sulphates"            "r.alcohol"             
[109] "w.alcohol"              "r.fi.fi"                "w.fi.fi"               
[112] "r.fi.vo"                "w.fi.vo"                "r.fi.ci"               
[115] "w.fi.ci"                "r.fi.re"                "w.fi.re"               
[118] "r.fi.ch"                "w.fi.ch"                "r.fi.to"               
[121] "w.fi.to"                "r.fi.pH"                "w.fi.pH"               
[124] "r.fi.su"                "w.fi.su"                "r.fi.al"               
[127] "w.fi.al"                "r.vo.fi"                "w.vo.fi"               
[130] "r.vo.vo"                "w.vo.vo"                "r.vo.ci"               
[133] "w.vo.ci"                "r.vo.re"                "w.vo.re"               
[136] "r.vo.ch"                "w.vo.ch"                "r.vo.to"               
[139] "w.vo.to"                "r.vo.pH"                "w.vo.pH"               
[142] "r.vo.su"                "w.vo.su"                "r.vo.al"               
[145] "w.vo.al"                "r.ci.fi"                "w.ci.fi"               
[148] "r.ci.vo"                "w.ci.vo"                "r.ci.ci"               
[151] "w.ci.ci"                "r.ci.re"                "w.ci.re"               
[154] "r.ci.ch"                "w.ci.ch"                "r.ci.to"               
[157] "w.ci.to"                "r.ci.pH"                "w.ci.pH"               
[160] "r.ci.su"                "w.ci.su"                "r.ci.al"               
[163] "w.ci.al"                "r.re.fi"                "w.re.fi"               
[166] "r.re.vo"                "w.re.vo"                "r.re.ci"               
[169] "w.re.ci"                "r.re.re"                "w.re.re"               
[172] "r.re.ch"                "w.re.ch"                "r.re.to"               
[175] "w.re.to"                "r.re.pH"                "w.re.pH"               
[178] "r.re.su"                "w.re.su"                "r.re.al"               
[181] "w.re.al"                "r.ch.fi"                "w.ch.fi"               
[184] "r.ch.vo"                "w.ch.vo"                "r.ch.ci"               
[187] "w.ch.ci"                "r.ch.re"                "w.ch.re"               
[190] "r.ch.ch"                "w.ch.ch"                "r.ch.to"               
[193] "w.ch.to"                "r.ch.pH"                "w.ch.pH"               
[196] "r.ch.su"                "w.ch.su"                "r.ch.al"               
[199] "w.ch.al"                "r.to.fi"                "w.to.fi"               
[202] "r.to.vo"                "w.to.vo"                "r.to.ci"               
[205] "w.to.ci"                "r.to.re"                "w.to.re"               
[208] "r.to.ch"                "w.to.ch"                "r.to.to"               
[211] "w.to.to"                "r.to.pH"                "w.to.pH"               
[214] "r.to.su"                "w.to.su"                "r.to.al"               
[217] "w.to.al"                "r.pH.fi"                "w.pH.fi"               
[220] "r.pH.vo"                "w.pH.vo"                "r.pH.ci"               
[223] "w.pH.ci"                "r.pH.re"                "w.pH.re"               
[226] "r.pH.ch"                "w.pH.ch"                "r.pH.to"               
[229] "w.pH.to"                "r.pH.pH"                "w.pH.pH"               
[232] "r.pH.su"                "w.pH.su"                "r.pH.al"               
[235] "w.pH.al"                "r.su.fi"                "w.su.fi"               
[238] "r.su.vo"                "w.su.vo"                "r.su.ci"               
[241] "w.su.ci"                "r.su.re"                "w.su.re"               
[244] "r.su.ch"                "w.su.ch"                "r.su.to"               
[247] "w.su.to"                "r.su.pH"                "w.su.pH"               
[250] "r.su.su"                "w.su.su"                "r.su.al"               
[253] "w.su.al"                "r.al.fi"                "w.al.fi"               
[256] "r.al.vo"                "w.al.vo"                "r.al.ci"               
[259] "w.al.ci"                "r.al.re"                "w.al.re"               
[262] "r.al.ch"                "w.al.ch"                "r.al.to"               
[265] "w.al.to"                "r.al.pH"                "w.al.pH"               
[268] "r.al.su"                "w.al.su"                "r.al.al"               
[271] "w.al.al"               

Now let’s remove all the columns that don’t start with r. or w.

dataX.nonlinear.final <- dataX.nonlinear[c(92:271)]
names(dataX.nonlinear.final)

  [1] "r.fixed.acidity"        "w.fixed.acidity"        "r.volatile.acidity"    
  [4] "w.volatile.acidity"     "r.citric.acid"          "w.citric.acid"         
  [7] "r.residual.sugar"       "w.residual.sugar"       "r.chlorides"           
 [10] "w.chlorides"            "r.total.sulfur.dioxide" "w.total.sulfur.dioxide"
 [13] "r.pH"                   "w.pH"                   "r.sulphates"           
 [16] "w.sulphates"            "r.alcohol"              "w.alcohol"             
 [19] "r.fi.fi"                "w.fi.fi"                "r.fi.vo"               
 [22] "w.fi.vo"                "r.fi.ci"                "w.fi.ci"               
 [25] "r.fi.re"                "w.fi.re"                "r.fi.ch"               
 [28] "w.fi.ch"                "r.fi.to"                "w.fi.to"               
 [31] "r.fi.pH"                "w.fi.pH"                "r.fi.su"               
 [34] "w.fi.su"                "r.fi.al"                "w.fi.al"               
 [37] "r.vo.fi"                "w.vo.fi"                "r.vo.vo"               
 [40] "w.vo.vo"                "r.vo.ci"                "w.vo.ci"               
 [43] "r.vo.re"                "w.vo.re"                "r.vo.ch"               
 [46] "w.vo.ch"                "r.vo.to"                "w.vo.to"               
 [49] "r.vo.pH"                "w.vo.pH"                "r.vo.su"               
 [52] "w.vo.su"                "r.vo.al"                "w.vo.al"               
 [55] "r.ci.fi"                "w.ci.fi"                "r.ci.vo"               
 [58] "w.ci.vo"                "r.ci.ci"                "w.ci.ci"               
 [61] "r.ci.re"                "w.ci.re"                "r.ci.ch"               
 [64] "w.ci.ch"                "r.ci.to"                "w.ci.to"               
 [67] "r.ci.pH"                "w.ci.pH"                "r.ci.su"               
 [70] "w.ci.su"                "r.ci.al"                "w.ci.al"               
 [73] "r.re.fi"                "w.re.fi"                "r.re.vo"               
 [76] "w.re.vo"                "r.re.ci"                "w.re.ci"               
 [79] "r.re.re"                "w.re.re"                "r.re.ch"               
 [82] "w.re.ch"                "r.re.to"                "w.re.to"               
 [85] "r.re.pH"                "w.re.pH"                "r.re.su"               
 [88] "w.re.su"                "r.re.al"                "w.re.al"               
 [91] "r.ch.fi"                "w.ch.fi"                "r.ch.vo"               
 [94] "w.ch.vo"                "r.ch.ci"                "w.ch.ci"               
 [97] "r.ch.re"                "w.ch.re"                "r.ch.ch"               
[100] "w.ch.ch"                "r.ch.to"                "w.ch.to"               
[103] "r.ch.pH"                "w.ch.pH"                "r.ch.su"               
[106] "w.ch.su"                "r.ch.al"                "w.ch.al"               
[109] "r.to.fi"                "w.to.fi"                "r.to.vo"               
[112] "w.to.vo"                "r.to.ci"                "w.to.ci"               
[115] "r.to.re"                "w.to.re"                "r.to.ch"               
[118] "w.to.ch"                "r.to.to"                "w.to.to"               
[121] "r.to.pH"                "w.to.pH"                "r.to.su"               
[124] "w.to.su"                "r.to.al"                "w.to.al"               
[127] "r.pH.fi"                "w.pH.fi"                "r.pH.vo"               
[130] "w.pH.vo"                "r.pH.ci"                "w.pH.ci"               
[133] "r.pH.re"                "w.pH.re"                "r.pH.ch"               
[136] "w.pH.ch"                "r.pH.to"                "w.pH.to"               
[139] "r.pH.pH"                "w.pH.pH"                "r.pH.su"               
[142] "w.pH.su"                "r.pH.al"                "w.pH.al"               
[145] "r.su.fi"                "w.su.fi"                "r.su.vo"               
[148] "w.su.vo"                "r.su.ci"                "w.su.ci"               
[151] "r.su.re"                "w.su.re"                "r.su.ch"               
[154] "w.su.ch"                "r.su.to"                "w.su.to"               
[157] "r.su.pH"                "w.su.pH"                "r.su.su"               
[160] "w.su.su"                "r.su.al"                "w.su.al"               
[163] "r.al.fi"                "w.al.fi"                "r.al.vo"               
[166] "w.al.vo"                "r.al.ci"                "w.al.ci"               
[169] "r.al.re"                "w.al.re"                "r.al.ch"               
[172] "w.al.ch"                "r.al.to"                "w.al.to"               
[175] "r.al.pH"                "w.al.pH"                "r.al.su"               
[178] "w.al.su"                "r.al.al"                "w.al.al"               

Now we’re ready to fit the data with a model. We no longer need the prior weights because the red and white wines now live in different dimensions of the data.

dataX.nonlinear.test <- dataX.nonlinear.final[c(test.indices),]
dataX.nonlinear.train <- dataX.nonlinear.final[-c(test.indices),]
set.seed(123)
model.nonlinear <- plsRglm(dataY.train,dataX.nonlinear.train,nt=10,limQ2set=.0975,
                       dataPredictY=dataX.nonlinear.train,modele="pls-glm-polr",family=NULL,typeVC="none",
                       EstimXNA=FALSE,scaleX=TRUE,scaleY=NULL,pvals.expli=FALSE,
                       alpha.pvals.expli=.05,MClassed=FALSE,tol_Xi=10^(-12),
                       sparse=FALSE,sparseStop=TRUE,naive=FALSE,verbose=TRUE)

____************************************************____
Only naive DoF can be used with PLS GLM

Model: pls-glm-polr 
Method: logistic 

____Component____ 1 ____
____Component____ 2 ____
____Component____ 3 ____
____Component____ 4 ____
____Component____ 5 ____
____Component____ 6 ____
____Component____ 7 ____
____Component____ 8 ____
____Component____ 9 ____
____Component____ 10 ____
____Predicting X without NA neither in X nor in Y____
****________________________________________________****

model.nonlinear

Number of required components:
[1] 10
Number of successfully computed components:
[1] 10
Coefficients:
                                [,1]
3|4                     0.6581066007
4|5                     2.8069273439
5|6                     5.9639681915
6|7                     8.6308236355
7|8                    11.0496809584
8|9                    15.1074411771
r.fixed.acidity        -6.4991486621
w.fixed.acidity         3.1757588445
r.volatile.acidity     -1.1204206789
w.volatile.acidity     -0.1173085603
r.citric.acid          -0.4850643745
w.citric.acid           0.5493441696
r.residual.sugar       -0.4098775709
w.residual.sugar       -0.0946582713
r.chlorides            -0.3238003724
w.chlorides             0.1284522445
r.total.sulfur.dioxide -0.0195994713
w.total.sulfur.dioxide  0.0464197462
r.pH                   -0.5566304720
w.pH                    0.5589080659
r.sulphates             0.0028262465
w.sulphates             0.3083566706
r.alcohol              -0.3079347055
w.alcohol               0.0040333633
r.fi.fi                19.0799599790
w.fi.fi                 3.8598794104
r.fi.vo                 0.3842029687
w.fi.vo                -0.5126331783
r.fi.ci                -0.5421993053
w.fi.ci                -6.1741461029
r.fi.re                 0.3218962198
w.fi.re                 2.6585273010
r.fi.ch                 0.4782511508
w.fi.ch                -0.1939124458
r.fi.to                 0.0710756054
w.fi.to                 0.0988255689
r.fi.pH                 0.5802452150
w.fi.pH                 0.1209011834
r.fi.su                 0.4948137929
w.fi.su                 0.5092107439
r.fi.al                -0.0507531332
w.fi.al                -0.5545008159
r.vo.fi                 0.3842029687
w.vo.fi                -0.5126331783
r.vo.vo                 0.1170197299
w.vo.vo                -0.0385453479
r.vo.ci                 0.3438101350
w.vo.ci                -0.1219763557
r.vo.re                -0.0027208287
w.vo.re                -0.4025007914
r.vo.ch                -0.0433883320
w.vo.ch                 0.0384529478
r.vo.to                 0.0268454217
w.vo.to                -0.0267501369
r.vo.pH                 0.0049481418
w.vo.pH                -0.0531237124
r.vo.su                -0.0961405848
w.vo.su                -0.0039936237
r.vo.al                -0.0468435522
w.vo.al                -0.0005152789
r.ci.fi                -0.5421993053
w.ci.fi                -6.1741461029
r.ci.vo                 0.3438101350
w.ci.vo                -0.1219763557
r.ci.ci                 0.5057359268
w.ci.ci                 0.7795513101
r.ci.re                 0.4001478801
w.ci.re                 0.6037981814
r.ci.ch                -0.1493637136
w.ci.ch                 0.0535397468
r.ci.to                 0.0980175319
w.ci.to                 0.0137857944
r.ci.pH                 0.2304521797
w.ci.pH                -0.1051006471
r.ci.su                -0.1180163286
w.ci.su                -0.0802514327
r.ci.al                -0.0661453037
w.ci.al                -0.0589079435
r.re.fi                 0.3218962198
w.re.fi                 2.6585273010
r.re.vo                -0.0027208287
w.re.vo                -0.4025007914
r.re.ci                 0.4001478801
w.re.ci                 0.6037981814
r.re.re                 0.2272403007
w.re.re                 0.1665229157
r.re.ch                 0.0394333828
w.re.ch                -0.1074187916
r.re.to                 0.0339822809
w.re.to                 0.0259064587
r.re.pH                 0.0179904207
w.re.pH                 0.1197861532
r.re.su                -0.2167001237
w.re.su                -0.4459108286
r.re.al                -0.0230056805
w.re.al                 0.0282968045
r.ch.fi                 0.4782511508
w.ch.fi                -0.1939124458
r.ch.vo                -0.0433883320
w.ch.vo                 0.0384529478
r.ch.ci                -0.1493637136
w.ch.ci                 0.0535397468
r.ch.re                 0.0394333828
w.ch.re                -0.1074187916
r.ch.ch                -0.0023186076
w.ch.ch                 0.0032009357
r.ch.to                -0.0053942521
w.ch.to                -0.0072973610
r.ch.pH                 0.0288025786
w.ch.pH                -0.0273572260
r.ch.su                 0.0407031161
w.ch.su                 0.0394381122
r.ch.al                -0.0070940742
w.ch.al                -0.0125756885
r.to.fi                 0.0710756054
w.to.fi                 0.0988255689
r.to.vo                 0.0268454217
w.to.vo                -0.0267501369
r.to.ci                 0.0980175319
w.to.ci                 0.0137857944
r.to.re                 0.0339822809
w.to.re                 0.0259064587
r.to.ch                -0.0053942521
w.to.ch                -0.0072973610
r.to.to                 0.0033288356
w.to.to                 0.0057165686
r.to.pH                 0.0036133358
w.to.pH                 0.0064476579
r.to.su                -0.0440206016
w.to.su                -0.0131449260
r.to.al                -0.0012642612
w.to.al                -0.0054600617
r.pH.fi                 0.5802452150
w.pH.fi                 0.1209011834
r.pH.vo                 0.0049481418
w.pH.vo                -0.0531237124
r.pH.ci                 0.2304521797
w.pH.ci                -0.1051006471
r.pH.re                 0.0179904207
w.pH.re                 0.1197861532
r.pH.ch                 0.0288025786
w.pH.ch                -0.0273572260
r.pH.to                 0.0036133358
w.pH.to                 0.0064476579
r.pH.pH                 0.0704027598
w.pH.pH                 0.0021288275
r.pH.su                 0.1881574061
w.pH.su                 0.0674640554
r.pH.al                -0.0128353131
w.pH.al                -0.0410647627
r.su.fi                 0.4948137929
w.su.fi                 0.5092107439
r.su.vo                -0.0961405848
w.su.vo                -0.0039936237
r.su.ci                -0.1180163286
w.su.ci                -0.0802514327
r.su.re                -0.2167001237
w.su.re                -0.4459108286
r.su.ch                 0.0407031161
w.su.ch                 0.0394381122
r.su.to                -0.0440206016
w.su.to                -0.0131449260
r.su.pH                 0.1881574061
w.su.pH                 0.0674640554
r.su.su                 0.2569719690
w.su.su                 0.0112073044
r.su.al                 0.0192420661
w.su.al                -0.0009256702
r.al.fi                -0.0507531332
w.al.fi                -0.5545008159
r.al.vo                -0.0468435522
w.al.vo                -0.0005152789
r.al.ci                -0.0661453037
w.al.ci                -0.0589079435
r.al.re                -0.0230056805
w.al.re                 0.0282968045
r.al.ch                -0.0070940742
w.al.ch                -0.0125756885
r.al.to                -0.0012642612
w.al.to                -0.0054600617
r.al.pH                -0.0128353131
w.al.pH                -0.0410647627
r.al.su                 0.0192420661
w.al.su                -0.0009256702
r.al.al                -0.0066022795
w.al.al                -0.0096352398
Information criteria and Fit statistics:
                AIC      BIC Missclassed Chi2_Pearson_Y
Nb_Comp_0  13287.31 13326.65        2928       6708.064
Nb_Comp_1  13162.59 13208.48        2908       6463.251
Nb_Comp_2  11844.55 11897.00        2526       5082.886
Nb_Comp_3  11514.18 11573.18        2423       4732.986
Nb_Comp_4  11403.17 11468.72        2372       4649.119
Nb_Comp_5  11354.12 11426.23        2348       4640.854
Nb_Comp_6  11344.02 11422.69        2333       4629.086
Nb_Comp_7  11333.53 11418.75        2359       4616.520
Nb_Comp_8  11301.54 11393.32        2357       4582.407
Nb_Comp_9  11247.62 11345.95        2380       4549.792
Nb_Comp_10 11221.78 11326.67        2379       4533.511

Now let’s see how well it did.

predictions.train <- as.numeric(predict(model.nonlinear$FinalModel))
observations.train <- as.numeric(dataY.train)
predictions.test <- as.numeric(predict(model.nonlinear, newdata = dataX.nonlinear.test, type="class"))
observations.test <- as.numeric(dataY.test)
mae(observations.train, predictions.train)

[1] 0.5086605

mae(observations.test, predictions.test)

[1] 0.4953846

The MAE has improved slightly, by about 1%. Let’s try this again, but restricted to the variables which showed the highest correlation with quality.

for (feature in names(dataX.nonlinear.train)) {
  feature.cor <- cor(dataX.nonlinear.train[feature], as.numeric(dataY.train))
  if (abs(feature.cor) > 0.2) {
    print(feature)
    print(feature.cor)
  }
}

[1] "w.fi.ch"
            [,1]
w.fi.ch 0.201826
[1] "w.fi.al"
             [,1]
w.fi.al 0.2183314
[1] "w.vo.ch"
             [,1]
w.vo.ch 0.2208955
[1] "w.vo.al"
             [,1]
w.vo.al 0.2374277
[1] "w.ch.fi"
            [,1]
w.ch.fi 0.201826
[1] "w.ch.vo"
             [,1]
w.ch.vo 0.2208955
[1] "w.ch.ch"
             [,1]
w.ch.ch 0.2257812
[1] "w.ch.al"
             [,1]
w.ch.al 0.2562421
[1] "w.pH.al"
             [,1]
w.pH.al 0.2063902
[1] "w.al.fi"
             [,1]
w.al.fi 0.2183314
[1] "w.al.vo"
             [,1]
w.al.vo 0.2374277
[1] "w.al.ch"
             [,1]
w.al.ch 0.2562421
[1] "w.al.pH"
             [,1]
w.al.pH 0.2063902
[1] "w.al.al"
             [,1]
w.al.al 0.2699345

These are the best correlations we see. They are very low and I don’t believe they will be useful. Plus, they are all white wine features.

Binning into extra dimensions

Let’s try another approach. We can bin the features, and let every bin become its own feature. This will create a new dimension, and thus a different fitted slope, for every bin, thus allowing nonlinear features to be described by the linear model.

We now transform the data points row by row into a more sparse dataset of binned features.

dataX.expanded <- data.frame(dataX)

There were 24 warnings (use warnings() to see them)

nbins.default <- 20
nbins <- nbins.default
for (feature in names(dataX.expanded)) {
  if (feature == "color") {
    nbins <- 2
  } else {
    nbins <- nbins.default
  }
  # map value of feature to bin number
  xmin <- 0.99*min(dataX.expanded[feature])
  xmax <- 1.01*max(dataX.expanded[feature])
  slope <- nbins/(xmax - xmin)
  feature.bin <- paste(feature, "bin", sep="")
  dataX.expanded[feature.bin] <- with(dataX.expanded, floor( slope*( eval(parse(text=feature)) - xmin) ) )
  
  # create new feature for each bin
  for (i in 1:nbins) {
    new.feature <- paste(feature, as.character(i), sep="")
    
    #dataX_expanded[new.feature] <- with(dataX_expanded, 
    #                      ifelse(eval(parse(text=feature)) < i/slope+xmin & eval(parse(text=feature)) > (i-1)/slope+xmin,
    #                              eval(parse(text=feature)) - xmin - eval(parse(text=feature.bin))/slope,
    #                              0.0
    #                       )
    #)
    
    #dataX_expanded[new.feature] <- with(dataX_expanded, 
    #                                    ifelse(eval(parse(text=feature)) < i/slope+xmin & eval(parse(text=feature)) > (i-1)/slope+xmin,
    #                                           eval(parse(text=feature)),
    #                                           0.0
    #                                    )
    #)
    
    dataX.expanded[new.feature] <- with(dataX.expanded, 
                                        ifelse(eval(parse(text=feature)) < i/slope+xmin & eval(parse(text=feature)) > (i-1)/slope+xmin,
                                               1,
                                               0.0
                                        )
    )
  }
}
dataX.expanded

We then remove all the extra columns and just keep x1, x2, …, x10 for each feature x.

dataX.binned <- dataX.expanded[,grepl(regex("\\d$"),names(dataX.expanded))]
dataX.binned

We also need to remove the columns that contain only zeroes, because they cause problems in the application of the model.

means <- sapply(dataX.binned, mean)
dataX.final <- dataX.binned[which(means > 0)]
dataX.final

Now we are ready to fit the data with the plsRglm model.

set.seed(123)
model.pls.expanded <- plsRglm(dataY,dataX.final,nt=10,dataPredictY=dataX.final,modele="pls-glm-polr")

____************************************************____

Model: pls-glm-polr 
Method: logistic 

____Component____ 1 ____

glm.fit: fitted probabilities numerically 0 or 1 occurred

____Component____ 2 ____

glm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurred

____Component____ 3 ____

glm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurred

____Component____ 4 ____

glm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurred

____Component____ 5 ____

glm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurred

____Component____ 6 ____

glm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurred

____Component____ 7 ____

glm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurred

____Component____ 8 ____

glm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurred

____Component____ 9 ____

glm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurredglm.fit: fitted probabilities numerically 0 or 1 occurred

____Component____ 10 ____
____Predicting X without NA neither in X nor in Y____
****________________________________________________****

model.pls.expanded

Number of required components:
[1] 10
Number of successfully computed components:
[1] 10
Coefficients:
                                [,1]
3|4                    -6.494061e+00
4|5                    -4.233340e+00
5|6                    -9.699601e-01
6|7                     1.770676e+00
7|8                     4.191299e+00
8|9                     7.985257e+00
fixed.acidity1         -5.182182e-01
fixed.acidity2         -1.129911e-01
fixed.acidity3         -5.914274e-02
fixed.acidity4         -1.154156e-01
fixed.acidity5          2.862123e-01
fixed.acidity6          3.389757e-01
fixed.acidity7          1.435659e-01
fixed.acidity8         -1.727532e-01
fixed.acidity9         -4.097428e-02
fixed.acidity10         9.025562e-03
fixed.acidity11         5.920507e-02
fixed.acidity12        -4.415641e-02
fixed.acidity13        -7.125334e-01
fixed.acidity14        -5.827418e-01
fixed.acidity15        -2.832894e-01
fixed.acidity16         1.186239e+00
fixed.acidity17        -1.863217e+00
fixed.acidity18         1.805096e+00
fixed.acidity20        -1.970031e+00
volatile.acidity1       4.077460e+00
volatile.acidity2       2.587004e+00
volatile.acidity3       1.333171e+00
volatile.acidity4       1.165663e+00
volatile.acidity5       7.331239e-01
volatile.acidity6       4.145725e-01
volatile.acidity7       1.082460e-01
volatile.acidity8       1.034790e-01
volatile.acidity9      -1.702506e-01
volatile.acidity10     -4.364826e-01
volatile.acidity11     -7.016081e-01
volatile.acidity12     -1.027963e+00
volatile.acidity13     -1.169550e+00
volatile.acidity14     -8.724708e-01
volatile.acidity15     -1.264333e+00
volatile.acidity16     -1.398794e+00
volatile.acidity17     -1.969510e+00
volatile.acidity18     -2.091438e+00
volatile.acidity19     -4.466069e-01
volatile.acidity20     -1.052737e+00
citric.acid1            1.780439e-01
citric.acid2            1.269171e-01
citric.acid3            2.124994e-01
citric.acid4            2.694033e-01
citric.acid5            2.257149e-01
citric.acid6           -1.078736e-01
citric.acid7           -2.483120e-01
citric.acid8           -2.403205e-02
citric.acid9            1.009071e-01
citric.acid10          -3.648438e-03
citric.acid11           3.838162e-01
citric.acid12           3.640878e-01
citric.acid13           7.298252e-01
citric.acid14           1.808723e+00
citric.acid15          -1.419474e-01
citric.acid17           2.651186e+00
citric.acid20           1.931306e+00
residual.sugar1         3.343470e+00
residual.sugar2         1.616747e+00
residual.sugar3         9.892573e-01
residual.sugar4         4.956498e-01
residual.sugar5         2.493510e-01
residual.sugar6         9.405254e-02
residual.sugar7        -3.458010e-02
residual.sugar8        -2.804988e-02
residual.sugar9         6.852988e-02
residual.sugar10       -1.227755e-01
residual.sugar11       -1.061652e-01
residual.sugar12       -1.719673e-02
residual.sugar13       -1.064788e-01
residual.sugar14       -1.556961e-01
residual.sugar15       -2.633151e-01
residual.sugar16       -7.877038e-01
residual.sugar17       -1.585952e-01
residual.sugar18        3.253778e-01
residual.sugar19        6.477780e-01
residual.sugar20       -2.163893e+00
chlorides1              2.378185e-01
chlorides2              2.100432e-01
chlorides3              1.717226e-02
chlorides4              3.787419e-01
chlorides5              1.733989e-01
chlorides6              1.343740e-01
chlorides7              1.763120e-01
chlorides8              1.171058e-02
chlorides9             -3.502257e-01
chlorides10            -3.088657e-01
chlorides11            -4.479072e-01
chlorides12            -1.561645e-01
chlorides13            -7.557955e-01
chlorides14            -3.708093e-01
chlorides15            -1.045747e-02
chlorides16            -7.679959e-01
chlorides17            -1.030835e+00
chlorides20             3.114531e+00
total.sulfur.dioxide1  -4.189395e-03
total.sulfur.dioxide2  -2.005031e-01
total.sulfur.dioxide3  -4.664131e-03
total.sulfur.dioxide4  -2.968508e-01
total.sulfur.dioxide5  -2.598868e-02
total.sulfur.dioxide6   4.671650e-01
total.sulfur.dioxide7   4.190414e-01
total.sulfur.dioxide8  -1.329759e-01
total.sulfur.dioxide9  -3.342838e-03
total.sulfur.dioxide10 -1.309128e-01
total.sulfur.dioxide11 -1.776151e-01
total.sulfur.dioxide12  1.289858e-01
total.sulfur.dioxide13  7.205884e-02
total.sulfur.dioxide14  1.278677e-01
total.sulfur.dioxide15  1.939871e-01
total.sulfur.dioxide16 -5.832429e-01
total.sulfur.dioxide17  4.969586e+00
total.sulfur.dioxide18  4.600023e+00
total.sulfur.dioxide20  5.479488e+03
pH1                     6.653837e-01
pH2                     1.453287e+00
pH3                     3.752981e-01
pH4                    -1.238611e-01
pH5                     1.547272e-01
pH6                     2.963736e-02
pH7                     1.409576e-03
pH8                     6.377656e-02
pH9                    -2.335318e-02
pH10                   -1.709131e-01
pH11                   -2.205734e-01
pH12                   -5.832958e-02
pH13                   -4.672400e-03
pH14                    4.915395e-01
pH15                    8.555796e-01
pH16                    1.377559e+00
pH17                   -5.272278e-01
pH18                    2.552647e+00
pH20                    2.115081e-01
sulphates1             -6.005279e-02
sulphates2             -3.320751e-01
sulphates3             -6.857051e-01
sulphates4             -7.132652e-01
sulphates5             -3.916924e-01
sulphates6             -2.718033e-01
sulphates7              5.937014e-02
sulphates8              1.392082e-01
sulphates9              1.607403e-01
sulphates10             2.225042e-01
sulphates11             2.510526e-01
sulphates12             5.486850e-01
sulphates13            -2.590564e-02
sulphates14             1.392501e-01
sulphates15             3.992617e-01
sulphates16             9.121835e-01
sulphates17            -1.764444e+00
sulphates18            -6.098982e-01
sulphates19             1.172595e+00
sulphates20             7.964866e-01
alcohol1                3.844442e+00
alcohol2                1.852312e+00
alcohol3                7.493680e-01
alcohol4                9.217644e-01
alcohol5                1.093438e+00
alcohol6                6.254462e-01
alcohol7                3.903187e-01
alcohol8                1.513303e-01
alcohol9               -3.627450e-01
alcohol10              -4.906415e-01
alcohol11              -1.021437e+00
alcohol12              -1.334117e+00
alcohol13              -1.551486e+00
alcohol14              -2.112469e+00
alcohol15              -2.336856e+00
alcohol16              -2.350504e+00
alcohol17              -2.429983e+00
alcohol18              -2.754132e+00
alcohol20               3.143425e+00
color1                 -3.104816e-01
color2                  3.104841e-01
Information criteria and Fit statistics:
                AIC      BIC Missclassed Chi2_Pearson_Y
Nb_Comp_0  16561.19 16601.86        3661       8388.609
Nb_Comp_1  14709.56 14757.02        3165       6314.419
Nb_Comp_2  14168.40 14222.63        3058       5836.756
Nb_Comp_3  13914.28 13975.29        2968       5610.210
Nb_Comp_4  13806.21 13874.00        2935       5527.397
Nb_Comp_5  13771.44 13846.01        2926       5493.560
Nb_Comp_6  13756.38 13837.73        2924       5477.910
Nb_Comp_7  13751.86 13839.98        2909       5471.762
Nb_Comp_8  13749.86 13844.77        2914       5464.127
Nb_Comp_9  13751.49 13853.17        2902       5461.685
Nb_Comp_10 13752.72 13861.18        2910       5464.299

Next, we evaluate the model’s performance like before.

predictions <- as.numeric(predict(model_pls_expanded$FinalModel))
observations <- as.numeric(dataY)
table(as.numeric(unlist(dataY)), predictions )

   predictions
       1    2    3    4    5    6
  1    1    4   16    9    0    0
  2    0    0  143   69    4    0
  3    0    6 1309  807   16    0
  4    0    0  622 1994  218    1
  5    0    0   33  765  279    2
  6    0    0    1  122   67    3
  7    0    0    0    1    4    0

plot(observations, predictions)

mae(observations, predictions)

[1] 0.4927648

This is the best MAE that I’ve seen so far, but I don’t think it’s good enough to justify the amount of effort we put into it and the long time it took to run it. It will also be difficult to expand the model to incorporate new data, since we removed many columns of zeros before fitting.

Conclusion

In conclusion, I would say that the most accurate and robust method that I have found is that of multiplying columns together to effectively turn the plsRglm model into a polynomial model. This method performed better than a simple plsRglm model and can easily be applied to new data. It could probably be improved further by engineering cubic and quartic terms, but the number of features would increase correspondingly and it would take much longer to run. I implemented the quadratic model in the script AFS_wine_quality_model_function.R. At this point, no ensemble method is applied.